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composing an image from the two inputted images based iq)on ihb abo> 
detemiined amount of the difference in focus. 

The computer program according to claiinji? whCTein the common in-focus 
area is user-adjustable. 

yf. The computer program accordipg to clainr^v wherein the focus difference 
determination further comprising: 

iteratively low-pass filterjil§ the common in-focus area of one of the inputted 
images; 

determining whptfier or not the low-pass filtered common in-focus area 
substantially matches the common in-focus area of another inputted images; and 

determhrfng the amount of the focus difference in the one common in-focus area 
between th^^o images based upon a number of the iterative low-pass filtering. 



REMARKS 



The Examiner rejected independent claims 1, 7 and 13 under S^^.S.C. § 102(b) 
as being anticipated by Kodama et al (1966). On the other hand^^the Examiner indicated 
the objection to claims 2 through 6, 8 through 12 and 14 thperfigh 16 as being dependent 
upon a rejected base claim, but would be allowable^if^ rewritten in independent form 
including all of the lunitations of the base claim ajKlany intervening claims. 

Newly amended independent cbrfns 1, 7 and 13 now each explicitly recite 
"determining a relative position bej^en two of the images based upon a predetermined 

set of movements " As sup^rted by the original disclosures at lines 7 through 18 on 

page 8 of the current appUcJation, the two images may be taken at a different focal length 
as well as with a diffemit optical axis as long as the two images contain the same objects. 
The relative positLem of these images is determined "based upon a predetermined set of 
movements" a?<explicitly recited by newly amended independent claims 1, 7 and 13. 
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In contrast to the above described feature, the cited reference discloses a set of / 
formula that describes how the desired all-focused image is related to multiple image^t 
a different focal length. The relationships are described in equations (1) through^. 
Although it is not explicitly stipulated, the images are taken at a focal lengthfe(ut along 
the same optical axis since no image transformation is disclosed between/(ne two images 
before the above equations are applied. / 

Based upon the above quoted patentable feature, the A;^licants respectfully 
submit to the Examiner that newly amended independent^ziaims 1, 7 and 13 are not 
anticipated by the cited reference. Similarly, dependenO^laims 2 through 6, 8 through 12 
and 14 through 16 ultimately depend from the newly amended independent claims and 
incorporate the patentable feature. Thus, the A^licant respectfully submit that the 
rejections of claims 1 through 17 should be )^thdrawn. 

Neyiy Added Claims 

The Applicants have added/claims 17 through 29 to the current application in the 
current response. Newly added independent claims 17, 22 and 27 have the combined 
subject matter limitations of independent claims 1, 7 and 13 as well as those of objected 
claims 2, 8 and 14. As the/Examiner indicated the allowable subject matter limitations, 
the Applicants respectfulfy submit that newly added independent claims ^,22 and 27 
should be allowable. /Since newly added dependent claims 18 through 21, 23 through 26, 
28 and 29 ultimately depend from newly added independent claims 22 and 27 and 
incorporated the/^lowable subject matter Umitations, newly added claims ^through^'^ 
should be patentable distinct. 

Tl^se claims are added without introducing new matter, and they have been 
supported by the original disclosures of the current application. The Applicants 
respectfully submit that the newly added claims should be entered and allowed. 
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Conclusion 



In view of the above amendments and the foregoina/i^marks, Applicant 
respectfully submits that all of the pending claims arejdxondition for allowance and 
respectfully request a favorable Office Action so iildicating. 



Respectfully submitted. 



Date: July 8, 2003 

KNOBLE & YOSHID^LLC 
Eight Penn Center, Sdite 1 350 
1628 John F. Kemjfedy Blvd. 
Philadelphia, P^l 9 103 
(215) 599-06C 




Ken I. Yoshida, Esq. 
Reg. No. 37,009 



8 



RCOH-1021/AP99-382 



PATENT 



I(x) = f(x) + g(x) 



then, 



\9 



5 I,(x) = f(x) + h2g(x) 

l2(x) = h,f{x) + g(x) 

where hi and h2 are blur functions. According to this model, eqi^ions (2) and (4) lead 
10 to 



G(x) = HI(x) 



and 



G(x) = (hrl)I,(x) + (hrl)l2l 



20 



25 




thus, the following equation is obtainei 



= h,h2-l 



For each input image, if the bllir functions hi and h2 are known, using the initially 
reconstructed image 10 = II /or 12. 

U = (H+l)In-G 



By iteration, the universally focused image is obtained. 

Similarl/ an arbitrarily focused image lab is expressed as follows when blur 
3 0 functions ha a/d hb are respectively used for a near image f(x) and a far image g(x). 
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Iab(x) = haf(x) + hbg(x) 



(9) 



Instead of Equation (8), 

Gab = (hbhi-ha) Ii(x) + (hahz-hb) I2(X)_ 



.(10) 



is considered 



10 



U, = (H+1) In-Gab.. 



-(11) 



r 
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for the iterative process for generating the/universally focused image. Furthermore, the 
Kodama et ah reference discloses two Ways to determine the blurring function. The first 
one uses predetermined image captupmg conditions including the focal point and at each 
distance, the blu rring function is mfeasured for an object. The second way is that the 
blu rring function is estimated byadjusting the blumess of each pixel as described in the 
Naito et al. reference. 

The above describecf two groups of prior art techniques are susceptible to errors in 
generating a universally focused image. In general, the conventional select and merge 
methods are subject to Selection errors, which cause an inferior merged image. The errors 
are particularly likely near edges where the intensity changes are larger in an out-of-focus 
image portion thanyan in-focus image portion. 

For the poor-art iterative reconstruction methods, in general, substantially no 
selection error^e involved. The iterative reconstruction methods do not need - 
information on the location of blumess, and it is possible to generate not only a 
universally /ocused image but also an arbitrarily focused image. On the other hand, it is 
not clear Wow many iterations are necessary to converge although the Kodama reference 
disclose/ only about three iterations. For a large number of iterations, an amount of 
calculation increases, and it takes a substantial amount of time. In addition, the two 
proposed techniques for determining the blurring functions also have the following 
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problems. For the first measurement technique, it is necessary to measure/rfie 
characteristics of the camera as well as other information on each inpu^mage such as the 
focal point length and exposure which affect the blurring function.yTor these reasons, 
images taken by an auto focus camera are not usually appropriaterfor the measurement 
5 technique since the auto focus camera generally does not provide information such as the 
focal point length. The other technique to determine the hturring function based upon the 
estimation from the image generally requires a large amount of calculation, and it takes a 
large amount of time. Furthermore, it is difficult focahe user to adjust the blurring 
function. / 

10 In addition to the above-described und^irable problems, the prior-art methods 

also have the following difficulties. Since tne zoom ratios usually change as the focal 
point length is changed, it is necessary to/correct at least the zoom ratio of each image 
before the portions of the images are nrerged into one composite image. For example, 
after an auto focus camera is positioned and captures a first image in which a 

1 5 predetermined object is centered^/o prepare for capturing a second image, the camera is 
turned away from the predeternfined object and the background scene is focused with the 
release button for exampl e. Aen, with the above described focus, the camera is turned 
back towards the predetermined object, and the second image is captured. The second 
image is centered arouncr the predetermined object. These two images have not only 

2 0 different zoom ratios but also slightly different orientations and positions. The ratio 
correction alone doe/ not allow the precise positioning of image portions. Images taken 
by a regular camera without a beam splitter and two CCD's require a high degree of . 
flexible re-positioning technique which includes parallel and rotational movements. 

In orde/ to solve the above-described problems, it is desired to quickly generate a 

2 5 high-quality pan focus composite image data from a plurality of common images captured 

by a generally available auto focus and auto exposure camera. To accomplish the above 
objective/, it is desired 1) to speed up the image composition process based upon the use 
of a bliping function, 2) to speed up the blurring function determination process, 3) to 
facilimte the confirmation and correction of the results by the blurring function, 4) to 

3 0 speculate the relative position of image data and to enable the position match for the 
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the relative position is acceptable in act 106. When the result is to be correcteji^gain, the 
preferred process goes back to the act 1 02. On the other hand, when the re^lt is not to be 
corrected or accepted, the preferred process goes to act 108, where an ip4bcus area is 
determined. / 
5 To determine the in-focus area, the act 108 relies upon the characteristic that the 

in-focus area has an increased amount of high-frequency regjOT than the out-of-focus 
area. The in-focus area determined in act 108 is correctednn act 1 10. Then, the preferred 
process determines an input blurring function in act 11^ and corrects the input blurring 
function in act 114 if desired. Similarly, the preferjped process determines an output 

1 0 blurring function in act 1 16. The act 1 16 outpuls an arbitrarily focused image by 
specifying an output result corresponding to Cna, hfe^ in equation (9). This act is not 
necessary when a universally focused image is to be outputted. The act 116 is similar to 
the act 1 14 in that an in-focus input image and a series of images produced by multiplying 
the input image by a set of blurring ftmctions. A user is requested to select one of the 

1 5 series of the images, and the blurring function (ha, hbb) used to generate the selected 

image is outputted. When an image without a blurring function is selected, a universally 
focused image is outputted. Based upon the above-determined data, the preferred process 
performs the composition of the images in act 118 before the composite image is 
outputted in act 120, / 

2 0 Using the deteitnined blurring functions (hi, h2, ha, hbb) from the acts 114 and 

1 1 6, the output of th/ composition process is an iterative reconstruction method. In 
general, the composition process is an iterative reconstruction method. Although the 
averaged pixel dstta after the two input images have been composed based upon the 
already deternimed relative position has some blur, the averaged pixel data is a good 

2 5 approximatioTi of the universally focused image. Based upon the above advantageous 

feature, th^average pixel data is used as an initial data set, and the equation (1 1) is 
reiterate^nly once in order to save a substantial amount of time. 

yThe initially reconstructed image is I^=(Il+I2)/2, and the equation (1 1) is 
performed once. After the above processing, the reconstructive image I^^I is expressed as 

3 0 foljjAs: 
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The above processing multiples each image data li and I2 by a filter inih2/2-h|hb+ha), 



(hih2/2-h2ha+hb) which is determined by a blurring function (hilwha^hb) and adding the 
products. The above universally focused image or the arbitrarfly focused image are 
outputted to output devices or storage such as a hard disk 210, a floppy disk 212 and a 
CRT display unit 204. / 

Now referring to FIGURE 2, a block diagrain illustrates one preferred 
embodiment of the image composition system according to the current invention. The 
preferred embodiment includes a central proc^sing unit CPU 200, a memory unit 202 
where software or programs run and various input/output devices are connected to the 
CPU 200 via a bus 201 . The input/outmift devices further include a cathode ray tube 
(CRT) display unit 204 with a displaVxontroller 203; input devices such as a keyboard 
207 and a mouse 208 with an inpurinterface 206; memory storage media such as a hard 
disk 210, a floppy disk 212 with4 floppy disk driver (FDD) 21 1 and a disk controller 209 
and an image input unit 214 vvith an input interface 213. For example, the software for 
composing the image mayyOe stored in a storage medium such as a floppy disk 212 and is 
read into the hard disk via the floppy disk drive 211. When the composition program is 
run, it is loaded into tUe memory 202. In addition, other storage media such as an optical 
medium with an oracal drive alternatively store the composition program. To input an 
image, one way i/to directly input a digital image captured by a digital camera via the 
input interface /l3. Another example is to scan in via the input interface213 an image 
printed from /silver halide film. Yet another example is to input an image from the 
floppy disl/2 12. 

Now referring to FIGURE 3, one exemplary software implementation shows a 
displa)^creen 204 where the act 104 of the preferred process is performed. A first image 
301 and a second image 302 are overlappingly displayed. The first image 301 is a 
%TOard while the second image 302 is the result of the processing. The display screen 
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204 includes a message or instruction 303 to a user, an O.K. button 304 for accepjifig the 
displayed image and a recalculate button 305 for initiating the calculation process. The 
user observes and determines whether or not the two images 301 and 302 correctly 
overlap. If the user determines that the two images correctly overlap, he or she clicks the 
5 O.K. button 304 via the mouse 208 of FIGURE 2. On the other han^ when the user 
determines that the two images do not correctly overlap, he or she can either manually 
move one image over the other via the mouse 208 or clicks the recalculate button 305 for 
automatically re-calculating the correct position. In any case, the user repeats either a 
combination of the above-described options until he or/he feels satisfied that the 

1 0 overlapping requirements k-are met. Upon satisfaction, he-the user clicks the O.K. button 
305, When the O.K. button 36 ^304 is clicked, the relative position of the two images 
301 and 302 is determined, and the preferred orocess proceeds to the act 108 of the flow 
chart as shown in FIGURE 1 . On the other Jiand, when the recalculate button 306 -305 is 
clicked, the relative position is recalculated in the act 102. 

1 5 Now referring to FIGURE 4, aii exemplary high-pass filter to determine an in- 

focus area is illustrated. The image mta is divided into blocks of 16 x 16 pixels. After 
processing with the high-pass filtOT having the coefficients as shown, each block is added. 
Assuming that Bij is the sum of pixel values in a block j from the image data i, (Blj - 
B2j) is larger as a high-frequency portion is larger in a first image 1 than second image 2. 

20 In other words, the first image 1 is brighter than the second image. The block having the 
highest value in B lj-B2/is considered as an in-focus image area of the first image 1 . 
Similarly, the block jJiaving the lowest value in Blj-B2j or having the highest value in 
B2j-Blj is considered as an in-focus image of the second image 2. 

FIGURE i shows an exemplary software implementation of the act 110 of the 

2 5 preferred process as shown in FIGURE 1 for accepting or correcting the above- 

determined in-focus image areas. The exemplary software implementation shows an 
image 501,ywhich is either the first image or the second image and an in-focus area 502. 
The exeniplary screen also includes the instruction 503 as well as an O.K. button 504. If 
the useryfletermines that the in-focus area is correct, he or she clicks the O.K. button 504. 

3 0 On theAther hand, if the displayed in-focus area is not acceptable, the user moves the box 
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502 to a correct in-focus area via the mouse 208 and clicks the O.K. button 504, XKe 
above-described operation is repeated for each of the two images. Based upon/tne above 
in-focus area as determined in the act 1 10, a blurring function is applied only to the in- 
focus area as described in the publication entitled "Enhanced Image Acgmsition by Using 
5 Multiple Differently Focused Images" by Naito et al. The applicatioi/of the blurring 
function produces gradually and sequentially Wurred images so tlm the in-focus area 
gradually matches with a corresponding out-of-focus area. iWgradual application 
determines the best matched blurring fiinction of the out-oWocused area. 

Now referring to FIGURE 6, a low-pass filter is used to gradually and iteratively 

1 0 blur an image, and one example of the low-pass filter/(s illustrated by a set of low-pass 
filter coefficients. In order to determine a blu rrinem inction for a second image data set 
with respect to a first image date -data set that Iws an in-focus area A, a corresponding out- 
of-focus area is designated as B in the seconja image data set. Using the above 
designations, the image data A after a firsrapplication of the low-pass filter is designated 

15 as A(l), and a second application to th/image data A(l) is designated as A(2). Similariy, 
after n-1 applications to the image Ma set A is designated as A(n-l), and after nth 
applications, the image data A is laesignated as A(n). The sum of a difference between 
A(n) and the corresponding B in absolute value for each pixel is designated as D(n). 
According to the above-described procedure, D(l), D(2), D(3) ... D(n) are sequentially 

2 0 determined, and the minimal value is sought among the above sums, n is a predetermined 

number. D(l), D(2) an(a D(3) gradually decrease towards D(n). D(n) is considered to be 
the minimal when D6n)>D(n+l) is confirmed. Thus, the corresponding image A(n) is 
considered to be the closest to the image B. As the low-pass filter as shown in FIGURE 6 
is applied to an image for n times, a blurring function is determined for the second image. 
25 Similarly, a blurring function for the first image is also determined in the above- 
described ni^ner. To determine a blurring function for the first image, an in-focus area 
of the seccmd image is gradually blurred to match a corresponding out-of-focus area of the 
first image. 

/Now referring to FIGURE 7, one exemplary software implementation shows a 

3 0 dismay screen where the act 1 14 of the preferred process is performed. The screen 



